This invention relates to an apparatus and a method for high fidelity reconstruction of an observed sample and for visual reconstruction of the data.
EMxe2x80x94electron microscopy
ETxe2x80x94electron tomography
CLxe2x80x94common lines method
F({overscore (X)})xe2x80x94density values at points (x,y,z) in a 3D reconstruction
m({overscore (X)})xe2x80x94a prior prejudice distribution
Cxe2x80x94A chi-squared statisticxe2x80x94a sum of squared differences between a projected 3D reconstruction and an observed projection divided by a measured variance of the observations.
W({overscore (X)})xe2x80x94an interpolation used for convolution operation given by the fact that a 3D density is built on a 3D grid, said grid values defining a continuous function through a trilinear interpolation and decreasing the resolution somewhat because of decreased bandwidth.
P(i)xe2x80x94a projection operation to the ith view
a pupil function based on the deflection of electrons by the iris diaphragm.
CTFxe2x80x94a contrast transfer function
PSFxe2x80x94a point spread function by which each projected image must be subjected to convolution. PSF in Fourier space is the pupil function * CTF.
T(i)({overscore (X)},{overscore (x)}(i))xe2x80x94a smearing function, based on the projected PSF
O(i)({overscore (X)},{overscore (Y)},{overscore (x)}(i))xe2x80x94an overlap function
*xe2x80x94multiplication
{circle around (x)}xe2x80x94convolution
In all data acquisition and signal processing systems resolution is a key factor. The degree of resolution is a direct measure of the size of the smallest detail that can be reproduced. The higher the resolution the better the recording. Resolution, however, is not all. A data acquisition system reacts, not only to the wanted quantity, but also to random processes such as noise and other interferences. When these interferences are of significant magnitude as compared with the resolution of the system, care must be taken in order to ensure that only meaningful information is extracted from the recorded data.
Increased resolution and filtering are common practice in data acquisition and processing. The basics of this can be found in any university textbook on the subject, and numerous inventions have been made in this field.
The present invention relates to data acquisition and processing systems of such dimensions that the sheer amount of data poses serious technical problems in implementing conventional techniques. A particular field where these problems occur is the field of three-dimensional (3D) imaging of small objects at high resolution. In this case, the mere size of the calculations needed presents substantial technical problems. This calls for the use of more refined methods. Once established, such methods could have a broad range of applications. Thus, the teaching from them could also be transferred to other dimensions, for instance 2D and 1D applications. Also, 3D imaging of large sized objects at high resolution is an interesting application.
The fundamentals of for instance 3D reconstruction were investigated in the beginning of this century mathematically by Johan Radon. The idea of 3D imaging of small objects (molecules) was shown and conceived by Aaron Klug and coworkers, but was not implemented for single particles.
Several techniques have since been developed for reconstructing 3D images of different types of objects. 3D imaging of macromolecular complexes lacking symmetry still has technical problems to be solved. In this case, computer power is not enough. The equations to be solved are of such proportions that earlier attempts to solve the technical problem of computing the 3D image reconstruction, from a series of recorded 2D projections from an electron microscope, so that it can be visually displayed as a 3D image, have had limited success due to the limited accuracy and high noise levels.
The technical problems concerned with the reconstruction of 3D images of small objects at high resolution from a number of 2D projections and with a high signal to noise ratio, have not yet been solved in prior art.
One technical problem to overcome is that of the contrast transfer function (CTF) of the electron microscope, or any other input means that is used. CTF is dependent on microscope and focus and most of the parameters are linked to the machine to be used. This makes it difficult to achieve a quantitatively correct reconstruction of the measured quantity.
The noise levels in 3D imaging have been too high for single particle molecular imaging. Using a larger set of tilts increases the risk of radiation damage to the specimen. Averaging has limited effect on artefacts in recorded data. These effects are reduced by averaging but are still present.
Gradient methods have been implemented and used more recently in electron microscopy (EM) 3D reconstructions. However, in EM applications, as well as in some medical radiation therapeutical applications, the use of gradient refinement methods for image restoration seldom results in a substantial improvement in the quality or resolution of the refined 3D reconstruction due to the fact that a large portion of data is missing, as the reconstruction is made from a limited tilt series.
Another technique of 3D reconstruction is to project onto convex sets, which utilises an envelope that engulfs the 3D reconstructed object. Density modulations outside the envelope are regarded as artefactual and are reset to a constant value during the iteration cycles. The iterative refinement cycles proceed until the density modulations become small. The degree of improvement in a 3D reconstruction provided with this method has not been unambiguously established by comparison with an objective model.
In the field of a 3D image reconstruction the resolution which it is desirable to obtain could require such a large number of recorded data sets that calculations could be difficult to accomplish and this thus sets a practical limit for the resolution. The use of symmetry of a crystalline specimen or internal symmetry of a specimen reduces the size of the calculations needed to a technically manageable level. However, not all of the objects in question have internal symmetry, crystalline symmetry, or symmetric arrangement, and it would therefore be advantageous to have a method of data acquisition and processing including 3D reconstruction, which is not dependent on symmetry. Prior art solutions have not yet overcome the noise problem from intermediate (below 15 xc3x85) to high (below 7 xc3x85) resolutions without the need for symmetry to manage the calculations.
A method of multi-scale reconstruction of the image of the structure of a body at an increased speed using iterative algorithms is disclosed in the U.S. Pat. No. 5,241,471. The algoriths are constructed such that the image of the object to be reconstructed is done on a structure, for instance a grid, having a finer and finer finesse from iteration to iteration. In this way the calculations could be done very fast. Thus, it will not work with fine details from the start and do not count on components having a resolution higher that the resolution of the structure (grid) actually in the iteration. No density distribution for individual grid points is done. The description of the sample on the grid is thus not changed from iteration to iteration, only its resolution.
An approach for providing a solution of this problem is to use maximum entropy. This is described by G. J. Ericksom and C. R. Smith, Maximum-Entropy and Bayesian methods in science and engineering, Volume 1:Foundations 1-314 (Kluwer Academic Publishers, Dordrecht, The Netherlands (1988)), by C. R. Smith and W. T. Grandy Jr, Maximum-Entropy and Bayesian methods in inverse problems, p 1-492 (D. Reidel Publishing Company, Dordrecht, Netherlands (1985)), and by B. Buck and V. A. Macaulay, 220 (Oxford University Press, New York (1991)). A maximum entropy 3D reconstruction has the property of being maximally noncommittal with regard to missing information (E. T. Jaynes, Physical Review 106, p 620-630 (1957)), and thus the maximum entropy method could serve as a powerful method to remove some of the detrimental effects caused by missing projections in ET reconstructions.
A method to perform a 3D reconstruction of an object with high resolution is described by Ali Mohammad-Djafari et al, xe2x80x9cMaximum Entropy Image Reconstructions in X-Ray and Diffraction Tomographyxe2x80x9d, IEEE Transactions on Medical Imaging Vol. 7 (December 1988) No 4,New York USA, PG. 345-354.
The argument and the applications in this referens refer only to a reconstruction from 1D to 2D, because of the idea that in medical imaging one reconstructs each circular xe2x80x9cslicexe2x80x9d independently from the others. Thus only 2D slice reconstructions are performed and the 3D is provided by adding the slices together. Thus the slices are treated as independent from each other. This method of treating the slices as independent gives unpredictable result when deconvoluting the real 3D point-spread function and possible line broadening. The result provided with the method described in this article can therefore not be quantitatively correct. When using the Poisson statistical properties of the image this is then not correct since it emanates from the a large part of the specimen, since the treatment of it is as if it were a local phenomenon restricted to a 1D line.
The implementations of the maximum entropy principles in ET 3D reconstructions have aimed at maximising the entropy while under the constraint that the reduced chi-squared statistic be equal to 1.0 as suggested by M. Barth, R. K. Bryan, R. Hegerl and W. Baumeister, Scanning Microsc. Suppl. 2, p 277-284 (1988) and by M. C. Lawrence, M. A. Jaffer and B. T. Sewell, Ultramicroscopy 31, p 285-301 (1989).
The usefulness of assigning low weight to improbable situations,so that the iterative solution of the problem becomes that of maximising the entropy relative to the non-informative prejudice, i.e. an estimated 3D prejudice distribution of the density, under the constraints of making the reduced chi-squared statistic equal to +1, has been shown by Gerard Bricogne in the article xe2x80x98Maximum entropy and the foundations of direct methodsxe2x80x99 in Acta Crystallographica A40, pp410-445 (1984), which discloses an algorithm used for constrained entropy maximisation for crystallographic phase refinement against reciprocal space data. However, the mathematics shown by G. Bricogne can not be applied on the tomographic problem. His method is a development of so called direct methods within crystallography which give a generalized solution of a statistical problem and does not deal with imaging problems. A diffraction pattern is provided which is the diffraction data from crystallographic examinations giving focal planes, and thus no image is provided. When several generated focal planes are collected, for instance on a photographic film or detector, then the phase information of the different arriving waves is lacking. The crystallographic problem is aimed at finding the phase information so that an image can then be calculated.
Thus the problem in crystallography is different to that of 3D imaging of macromolecular complexes lacking symmetry, which is solved by the invention. However, the tools that have been developed to be used for providing a high fidelity reconstruction in real space could be the knowledge that it is important to have a constrained maximum entropy formalism and to use Taylor expansion of the chi-squared statistic and the entropy as quadratic models.
This kind of direct method described by Gerard Bricogne is directed to solve the phase problem in crystallography and is therefore not applicable in 3D imaging of macromolecules lacldng symmetry. Therefore, the whole concept had to be totally redesigned in order to perform for the real space imaging problem. The algorithm used for constrained entropy maximisation was thus first devised for crystallographic phase refinement against reciprocal space data. It was adapted to the tomographic situation by replacing the Fourier transformation which relates molecular model to diffraction data in the crystallographic setting by line projections in real space. However, it could not handle projections at any angle. The problem was to provide a method which was independent of specific data sampling strategies like single-axis tilt series, conical tilt series, random tilt series, or tilts from symmetric samples where the projection angles could be calculated later (as is the case for the adenovirus). This technical problem has now been solved according to the invention.
Is is an object of the invention to provide high fidelity reconstruction of an observed sample, for instance by 3D imaging, practically independent of the size of the object.
It is an object of the invention to overcome the technical problems mentioned above, concerned with 3D imaging of single particle macromolecular complexes at high resolution, with a high signal to noise ratio.
It is a further object of the invention to provide a method and an apparatus which is independent of the object""s symmetry and which also offers the possibility of removing detrimental effects during the reconstruction of missing data.
It is also an object of the present invention to provide a 3D image reconstruction of single particle macromolecular objects lacking symmetry, with a high enough resolution to make it possible to reconstruct a 3D image of macromolecular objects lacking symmetry, using a series of 2D projections recorded with an electron microscope or a similar means.
It is a further object of the present invention, to solve the technical problem of the high noise levels present in 3D imaging, thus providing a low noise 3D imaging.
Another object of the invention, is to overcome the technical problem of the contrast transfer function of the electron microscope, or any other input means that is used, which makes it difficult to achieve a quantitatively correct reconstruction of the measured quantity.
A further object of the present invention. is to minimise the detrimental effects of non-recorded data on the reconstructed image.
Still a further object of the present invention, is to overcome the technical problem of imaging objects lacking symmetry.
Another object of the invention, is to overcome the technical problem of defining the physical parameters needed for accurate and optimal reconstruction.
Still another object of the invention is to provide an apparatus and a method which record, compute and present a true, i.e. a quantitatively correct, reconstruction of an observed sample with very high resolution.
Another object of the invention is to provide a reconstruction easy to use for the reconstruction of samples in different dimensions, for instance 3D, 2D and 1D appplications.
According to the invention a method and an apparatus is provided for generating a high fidelity reconstruction built on a grid of an observed sample, comprising: providing an initial estimated distribution of the sample built on the grid; providing a blurred prior prejudice distribution using estimated data; calculating in an iterative process for each iteration:
a new estimated distribution of the sample using comparison between the estimated distribution in the next preceding iteration and observed data of the sample,
a new prior prejudice distribution on the new estimate less blurred than the prior prejudice distribution in the next preceding iteration;
continuing the iterations until the difference between the new estimated distribution and the next preceding estimated distribution is less than a predetermined condition.
The blurring is preferably provided by making a Fourier transform of the estimated structure and by multiplying the coefficients of this Fourier transform with the Fourier coefficients of a gaussian and spherical filter. The blurred structure is normalised before it is used as the new prior prejudice distribution in the next iteration cycle. According to a prefered aspect of the invention the following method steps are done:
a) providing several recorded observed data of the sample, each from a different aspect of the sample;
b) providing a variance for individual observation grid points in each recorded observed data;
c) calculating the reconstruction of the observed sample structure in the iterative process taking said initial estimated distribution as a first iteration reconstruction approximation;
d) for each iteration: calculating said new prior prejudice distribution using the next preceding reconstruction result, calculating a reduced chi-squared statistic (C) using the immediately preceding calculated reconstruction, the recorded observations and said variance, and the grid, while maintaining normalization of the calculated reconstruction, calculating an entropy (S) using the immediately preceding calculated reconstruction and the new prior prejudice distribution;
e) maximising the entropy under the constraint of driving the reduced chi-squared statistic towards +1, and providing a new calculated reconstruction to use in the next iteration cycle; and
f) presenting the reconstruction.
Particularly, an apparatus and a method have been developed by which it is possible to record, compute and present a quantitatively correct 3D image of a macromolecular complex at high resolution on a visual display. However, the method as such is independent of size of the object and thus a correct 3D image of large sized objects could be done as well, for instance from tomographic recordings.
The invention is a development of, and thus based on, a maximum entropy method adapted to tomographic principles. According to the invention the Fourier transformation (which relates model to data in the crystallographic setting) is replaced by line projections in real space.
Thus, the maximum entropy distribution has, among other things, the property that no possibility is ignored, and that all possible situations are given a positive weight. A more powerful formulation than were proposed in the prior art is then to specifically make use of prior information in the algorithm and assign low weights to improbable situations, so that the iterative solution of the problem is that of maximising the entropy relative to the non-informative prior prejudice, i.e. an estimated 3D probability distribution of the density under the constraint that the reduced chi-squared statistic goes towards +1.
The maximum entropy 3D reconstruction has the property of being maximally noncommittal with regard to missing information, and could thus serve as a powerful method to remove some of the detrimental effects caused by the missing projections in ET (electron tomography) reconstructions. The implementations of the maximum entropy principles in ET 3D reconstructions have aimed at maximising the entropy under the constraint that reduced chi-squared statistic is aimed towards but does not go below +1. The reduced chi-squared statistic is the sum of the squared differences between the projected 3D reconstruction and the observed projections, each divided by the measured variances of the observations, and divided by the number of gridpoints of the projection in question. The variance of the observation data is the stated knowledge of them, and therefore the reduced chi-squared statistic is not allowed decrease below +1, because then that would mean that there are differences which are more relevant than the variance of the data. The maximum entropy distribution also has the property that no possibility is ignored and that all possible situations are given a positive weight.
The inventive method is preferably performed by having the same resolution and using the same grid for each projection. The reason for having low resolution of the density distribution on the grid points at the beginning and a high resolution later on is that it describes the probable density distribution of the objects in space. Thus, it does not describe that the object itself is blurred, but that the probable density distribution on the grid of it is blurred. After the first iteration a better adaptation has been done and because the entropy, for instance, has been higher the influence of the background has been lower. Therefore, a more narrow density distribution could be assumpted in the next iteration cycle. Thus, no finer finess regarding the grid is done as in the U.S. Pat. No. 5,241,471 mentioned above. In the invention instead, the blurring is changed from a starting density distribution to give a better and better density resolution at the grid points from iteration to iteration.
With an apparatus and a method according to the invention, it is possible to solve the technical problem of high resolution 3D imaging of macromolecular objects lacking symmetry by using a reconstruction procedure that exploits the prior information, thereby providing a model-free refinement, and producing a 3D structure whose projection convoluted with the point spread function of the microscope will adapt to the observed structure to give a low noise 3D reconstruction that is quantitatively correct.
Thus, the results provided by the invention gives more reliable results than refinements based on overparametrised models. For instance, in the case of using diffraction, if the physics for the model refinement in not correct, then it is very difficult to come to an adequate result. This difficulty is avoided according to the invention.
The key to solving these technical problems lies within the practical solving of mathematical equations. All prior attempts to reconstruct and visually display a true 3D image of single macromolecular complexes lacking symmetry have failed, due to the fact that equations technically solvable in practice have not been available.
The awesome size of the calculations needed is reduced considerably by the use of an approximation that the value of the smearing function is 30 only for a limited number of grid points, and by continuous improvement of the prejudice distribution, which reduces the amount of iteration cycles needed. Also a reduced calculation expression is given which speeds up the calculation such that it can be evaluated in about 1 minute, instead of about 300 years of evaluation time without the reduced expression. Also, a higher calculation precision is provided when using the reduced calculation expression. A further detailed description and discussion about this feature is given below in the description of the preferred embodiment.
An additional advantage of the invention is that it ends up close to the high theoretical resolution level given by Shannon""s law. This means that the method according to the invention could be applied to any conventional data acquisition and processing system, for instance an over-sampled such, where it would provide both a higher resolution, due to the noise eliminating qualities of the method, than is possible using prior art methods.
What has been further perceived according to the invention is that a multiplying by the coefficients of the Fourier transform of the PSF function and the projected 3D density must be done instead of a division of the Fourier coefficients of the observed image with the Fourier coefficients of the PSF function because it is by using such a division many of the errors in the previous attempts to achieve a true reconstruction have been introduced. Thus the invention creates a 3D-structure whose projections after blurring with the PSF function will be the same as the observed projections. Thus, when the iterations are ended then a 3D-structure is provided in which practically no errors have been introduced.